Skip to content

Python: Prevent divergence in type-hint analysis. (ODASA-8075)#1869

Merged
BekaValentine merged 4 commits into
github:masterfrom
tausbn:python-fix-typehint-divergence
Sep 6, 2019
Merged

Python: Prevent divergence in type-hint analysis. (ODASA-8075)#1869
BekaValentine merged 4 commits into
github:masterfrom
tausbn:python-fix-typehint-divergence

Conversation

@taus-semmle

@taus-semmle taus-semmle commented Sep 4, 2019

Copy link
Copy Markdown
Contributor

This fixes an unfortunate (but highly specific) divergence in the points-to analysis.
The following piece of code is a (more or less) minimal example that exhibits the divergence:

class foo():
    pass

while True:
    foo = foo[foo]

The culprit is the support for subscripted types originally introduced in #1728. Essentially, given the above, foo (the variable) can point to the subscripted type foo[foo], but then also the subscripted type (foo[foo])[(foo[foo]) and so on. Thankfully, only two projects on LGTM.com exhibited the above divergence.

To fix this, we block the recursion by requiring that the two types involved in a type subscripting operation (A and B in A[B]) must not themselves be subscripted types.

As this fixes an (obscure) regression in the analysis, I don't think it needs a change note.

Performance comparison here: https://git.semmle.com/gist/taus/8c8f31f2bb72055e50cc97bcce070747

Looks like a ~30% increase in performance across the board. Also seems to remove a bunch of false positives.

Otherwise it'll keep timing out until the fix has been pushed to LGTM.com
@taus-semmle taus-semmle marked this pull request as ready for review September 5, 2019 11:20
BekaValentine
BekaValentine previously approved these changes Sep 5, 2019

@BekaValentine BekaValentine left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable!

@taus-semmle

Copy link
Copy Markdown
Contributor Author

As discussed, I've added a test case to show what happens when nested types are encountered. My main worry was that something like List[List[int]] would be ignored by the analysis, since it contains a nesting (and thus not be recognised as a subscripted List type). This is not the case, however, as the tests show. Instead it gets recognised as a subscripted List type where the subscript is unknown. I think this is fine -- the main point of the subscript handling is to ensure that List[int] gets analysed as having the class List, and that'll continue to be the case even with the changes in this PR.

@BekaValentine BekaValentine merged commit 9eebe00 into github:master Sep 6, 2019
@tausbn tausbn deleted the python-fix-typehint-divergence branch February 12, 2021 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants